download_model.sh: prefer huggingface CLI when available#248
Conversation
The plain curl path 400s on the ~430 GB PRO bundle because Hugging Face's xet bridge rejects range-less GETs above a certain size. The official 'hf' (or 'huggingface-cli') client is xet-bridge native and already handles the chunked / ranged retrieval the bridge requires. Pick it up when on PATH; otherwise fall through to the existing curl flow unchanged. Token is passed via HF_TOKEN env rather than --token so it does not appear in 'ps' output of the child process.
|
Thanks @ivanfioravanti — your table in #249 was the clearest evidence of the size-threshold shape; I've linked it from the PR body so this closes it on merge. On the pure-curl concern: this patch keeps the existing curl flow bit-for-bit unchanged. The HF CLI is opt-in (detected at runtime, not required), and the patch is +16/-0 with the new branch nested inside That said, if @antirez would rather keep this script truly pure-curl (no HF detection at all), the alternative is a chunked |
|
Thanks @siraustin for the PR, I think actually both should be implemented so HF is used if available, while if not available curl can still download big models, what do you think? |
3 participants
Closes #249.
Problem
./download_model.sh pro-imatrixfails when fetching the new ~430 GB PRO bundle:Hugging Face's xet bridge rejects range-less GETs above a certain size threshold and returns a generic
<h1>400</h1>HTML page. Per #249's table, the threshold sits betweenq4-imatrix(164 GB, fine) andpro-imatrix(464 GB, 400). Empirically on the PRO file:curl --range 0-67108863(64 MB chunk) →206 Partial Content, downloads finecurlwith noRangeheader → 400 from the xet bridge after two redirectsThe redirect chain is HF resolve → xethub bridge → S3 presigned URL with
X-Xet-Cas-Uid; the bridge itself is what 400s, not S3.curl -C -doesn't help on a fresh download because there's no local partial to resume from. The smaller Flash files don't trip the threshold; PRO surfaces it.Fix
Prefer the official
hfCLI (orhuggingface-cliif it's the older binary on the user's PATH) — it is xet-bridge native and handles chunked / ranged retrieval correctly. Fall through to the existingcurlflow unchanged when no HF CLI is installed, so the existing Flash-download experience is preserved.The token is passed via the
HF_TOKENenv rather than--tokenso it doesn't appear inpsoutput of the child process.Test
mtp(small, cheap to interrupt) →hfselected,.incompletestaging file created, download begins; killed after a few seconds without side effects.pro-imatrix→ in-flight on a 512 GB M3 Ultra at time of writing, against this exact code path.curlcleanly when neitherhfnorhuggingface-cliis on PATH.Notes
command -v.curl --rangeloop that issues fixed-size chunks and concatenates — happy to take that direction if you'd rather not introduce the HF CLI as an option. The current patch is the smallest fix that works.